\appendix
\chapter{Source Selection and Analysis Method: Non-Functional Requirements for
Service-Based Applications}
\label{append:analysis}


We selected the sources proposed by \cite{Kitchenham08} for searching primary
studies. These sources contain the works published in journals, conferences and
workshops which are of recognized quality within the research community. The
search engines are: \textit{(i) IEEE Computer; (ii) ACM Digital Library;} and
\textit{(iii) Science Direct}. In the selected sources, we used the following search
query criteria:

\begin{center}
\texttt{(((``non functional properties'') OR (``non functional requirements''))
AND ``web service'' AND ``composition''))}
\end{center}

   
After define the sources' selection, we apply the
process used to identify those works that provided direct
evidence with regard to the research questions. Deciding
for the inclusion and exclusion criteria for filtering the corpus works
selection, we selected those related to non-functional requirements/properties,
and quality for web service based applications. Initially, the selection
criteria were interpreted liberally and clear exclusions were only made with
regard to title, abstract and introduction.

Based on the guidelines mentioned in \cite{Kitchenham08}, we established a
multi-step process made up of three steps with different selection
criteria:

\begin{itemize}
\item Step 1 - the search string must be run on the selected
search engine. An initial set of studies was obtained by filtering
of title, abstract, and if necessary, introduction. All the studies were
selected according to the inclusion and exclusion criteria. Studies which
were not clearly related to any aspect of the research questions
were excluded.
\item Step 2, the exclusion criteria were based on the following
practical issues: non-English papers, non-
International Conference papers and non-International Workshop
papers. Specifically in the case of ACM library, we considered only the
transaction journal works.
\item Step 3, the papers selection process was based on detailed
research questions.
\end{itemize}

The information for each step was collected considering the 3 searchers
and the the query presented previously. the results of each step were:
\textit{(i)} for each source a list of all the studies that
fulfilled the query; \textit{(ii)} a list
of studies for each source which contained all the works that did not fulfill the
second stage inclusion criteria; and
\textit{(iii)} the last step produced a list of works for each source which
contained all the studies that fulfilled the second step (table
\ref{tab:result01}).

 
\begin{table}
\begin{tabular}{l|c|c|c|c}
  \hline
  \hline
   & IEEE Explorer & ACM Library & Science Direct & Total \\
  \hline
  \hline
  Total results & 65 & 271 (75 \footnote{We considered only 75 transactions
  journal works}) & 166 & 502 (306 \footnote{considering the 75 ACM
  transactions journal works}) \\ 
  \hline
  Step one - results selected & 19 & 10 & 20  & 49 \\
  Step one - results selected (\%) & 29.23\% & 13.33\% & 12\% &
  16\% \\ 
  \hline 
  Step two - results selected & 7 & 3 & 9 & 19\\
  Step two - results selected (\%)  & 10.76\% & 4\% & 5.42\% & 6.20\%\\ 
  \hline
  \hline
\end{tabular}
\caption{Summary of the studies selected at each step.}
\label{tab:result01}
\end{table} 

 
The extraction of information was based on the research questions, and each work extraction question included the following
items: \textit{(i)} where the paper was found; \textit{(ii)} identification of the title and main
subjects; \textit{(iii)} summary of the research; \textit{(iv)} inclusion and
exclusion criteria; \textit{(v)} objective and result; and \textit{(vi)}
subjective results.

% The systematic literature review took place between December 2011 and January
% 2012 and we did not filter works within a specific year interval. Although the
% studies which were concretely analyzed were published between 2005 and 2011.
Table \ref{tab:result01} shows a summary of the studies selected in each stage
of the selection procedure for each source. The ``Total results'' were obtained
by running the search string on the selected sources. The next four rows show
the results obtained after applying stages one (2 first rows) and two (2 last
rows) of the studies selection procedure. 

In the first step, respecting the filters described, the 65 articles collected
from IEEE, only 29.23\% of them were in accordance with the criteria
described early, representing 19 articles. In ACM Library, from 75 works
collected, only 13.33\% passed in the first stage filter, representing 10 articles. In Science Direct
had the lowest percentage, totaling 20 of the 166 articles collected by the
query, thus representing 12\% of the total. Despite being the lowest relative
value, the Science Direct had the largest absolute result, with 20 works
in the first step. In the second stage, the percentage dropped further, and the
relevant works and with accordance to the criteria have been collected as the
final result. The results were respectively, 10.76\%, 4\% and 5.42\% of total
from the IEEE, ACM and Science Direct. The highest percentage was among the
 works from  IEEE, while the largest number od results, in absolute terms,
 was collected from Science Direct. The approaches resulting from this last
 stage were studied in depth and information concerning the detailed research
 questions and other fields of the extraction forms was extracted from each
 eork we selected. 49 works were selected in the first stage, and, only 19
 works in the second stage. It represents  6.20\% of the total amount of works.

Figure \ref{fig:statistics} shows the publications per year, from
2005 to 2011. 12 of 19 articles were selected in the systematic review published
in 2006, 2009 and 2010, being four in each year and 6 in Science Direct source,
5 in IEEE, and only 1 at the ACM. All nine of Science Direct were
published in the last 4 years. Figure also
shows that the number of publications that consider classification of NFR once
again increased from 2008. 

\begin{figure} 
\centering
\includegraphics[width=.6\textwidth]{chapters/state_ofthe_art/figs/data.png}
\caption{Publications per year.}
\label{fig:statistics}
\end{figure} 

